Terracotta: A tool for conducting experimental research on student learning.

For researchers seeking to improve education, a common goal is to identify teaching practices that have causal benefits in classroom settings. To test whether an instructional practice exerts a causal influence on an outcome measure, the most straightforward and compelling method is to conduct an experiment. While experimentation is common in laboratory studies of learning, experimentation is increasingly rare in classroom settings, and to date, researchers have argued it is prohibitively expensive and difficult to conduct experiments on education in situ. To address this challenge, we present Terracotta (Tool for Education Research with RAndomized COnTrolled TriAls), an open-source web application that integrates with a learning management system to provide a comprehensive experimental research platform within an online class site. Terracotta automates randomization, informed consent, experimental manipulation of different versions of learning activities, and export of de-identified research data. Here we describe these features, and the results of a live classroom demonstration study using Terracotta, a preregistered replication of McDaniel et al. (Journal of Applied Research in Memory and Cognition, 1(1), 18-26, 2012). Using Terracotta, we experimentally manipulated online review assignments so that consenting students alternated, on a weekly basis, between taking multiple-choice quizzes (retrieval practice) and reading answers to these quizzes (restudy). Students' performance on subsequent exams was significantly improved for items that had been in retrieval practice review assignments. This successful replication demonstrates that Terracotta can be used to experimentally manipulate consequential aspects of students' experiences in education settings.

Learning Introductory Biology: Students' Concept-Building Approaches Predict Transfer on Biology Exams.

Previous studies have found that students' concept-building approaches, identified a priori with a cognitive psychology laboratory task, are associated with student exam performances in chemistry classes. Abstraction learners (those who extract the principles underlying related examples) performed better than exemplar learners (those who focus on memorizing the training exemplars and responses) on transfer exam questions but not retention questions, after accounting for general ability. We extended these findings to introductory biology courses in which active-learning techniques were used to try to foster deep conceptual learning. Exams were constructed to contain both transfer and retention questions. Abstraction learners demonstrated better performance than exemplar learners on the transfer questions but not on the retention questions. These results were not moderated by indices of crystallized or fluid intelligence. Our central interpretation is that students identified as abstraction learners appear to construct a deep understanding of the concepts (presumably based on abstract underpinnings), thereby enabling them to apply and generalize the concepts to scenarios and instantiations not seen during instruction (transfer questions). By contrast, other students appear to base their representations on memorized instructed examples, leading to good performance on retention questions but not transfer questions.

Individual Differences in Structure Building: Impacts on Comprehension and Learning, Theoretical Underpinnings, and Support for Less Able Structure Builders.

In this article, we highlight an underappreciated individual difference: structure building. Structure building is integral to many everyday activities and involves creating coherent mental representations of conversations, texts, pictorial stories, and other events. People vary in this ability in a way not generally captured by other better known concepts and individual difference measures. Individuals with lower structure-building ability consistently perform worse on a range of comprehension and learning measures than do individuals with higher structure-building ability, both in the laboratory and in the classroom. Problems include a range of comprehension processes, including encoding factual content, inhibiting irrelevant information, and constructing a cohesive situation model of a text or conversation. Despite these problems, recent research is encouraging in that techniques to improve the learning outcomes for low-ability structure builders have been identified. We argue that the accumulated research warrants the recognition of structure building as an important individual difference in cognitive functioning and that additional theoretical work is needed to understand the underpinnings of structure-building deficits.

Effects of feature highlighting and causal explanations on category learning in a natural-science domain.

Teaching natural-science categories is highly challenging because the objects in such categories are composed of numerous complex dimensions that need to be perceived, evaluated, and integrated. Furthermore, the boundaries separating such categories are often fuzzy. A technique that has been proposed and investigated for enhancing the teaching of natural-science categories is feature highlighting, in which diagnostic features for identifying category members are explicitly described and illustrated. Using rock classification in geology as an example target domain, the present study further investigated the potential benefits of feature highlighting and also of providing causal explanations for the highlighted features. The authors found that feature highlighting did not always lead to improved generalization to novel members of the taught categories. However, robust beneficial effects were seen when the categories were relatively confusable ones and the stated diagnostic features were highly valid for distinguishing among the categories. Finally, at least under the present conditions, supplementing the highlighted features with causal explanations of the reasons for their occurrence did not further enhance the participants' rock-classification learning and generalization. Although the teaching of causal explanations is fundamental to science education, clear evidence that causal explanations enhance classification-learning per se in this domain remains to be demonstrated. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Using Students' Concept-building Tendencies to Better Characterize Average-Performing Student Learning and Problem-Solving Approaches in General Chemistry.

We previously reported that students' concept-building approaches, identified a priori using a cognitive psychology laboratory task, extend to learning complex science, technology, engineering, and mathematics topics. This prior study examined student performance in both general and organic chemistry at a select research institution, after accounting for preparation. We found that abstraction learners (defined cognitively as learning the theory underlying related examples) performed higher on course exams than exemplar learners (defined cognitively as learning by memorizing examples). In the present paper, we further examined this initial finding by studying a general chemistry course using a different pedagogical approach (process-oriented guided-inquiry learning) at an institution focused on health science majors, and then extended our studies via think-aloud interviews to probe the effect concept-building approaches have on problem-solving behaviors of average exam performance students. From interviews with students in the average-achieving group, using problems at three transfer levels, we found that: 1) abstraction learners outperformed exemplar learners at all problem levels; 2) abstraction learners relied on understanding and exemplar learners dominantly relied on an algorithm without understanding at all problem levels; and 3) both concept-building-approach students had weaknesses in their metacognitive monitoring accuracy skills, specifically their postperformance confidence level in their solution accuracy.

Testing Effect on High-Level Cognitive Skills.

The testing effect is one of the strongest learning techniques documented to date. Although the effects of testing on high-level learning are promising, fewer studies on this have been done. In this classroom application of the testing effect, we aimed to 1) determine whether a testing effect exists on high-level testing; 2) determine whether higher-level testing has an effect on low-level content retention; and 3) determine whether content knowledge, cognitive skill, or additional components are responsible for this effect. Through a series of two experiments, we confirmed a testing effect on high-level items. However, improved content retention due to testing was not observed. We suggest that this high-level testing effect is due to a better ability to apply specific skills to specific content when this application process has appeared on a previous exam.

Training Learning Strategies to Promote Self-Regulation and Transfer: The Knowledge, Belief, Commitment, and Planning Framework.

Surveys indicate that at all educational levels students often use relatively ineffective study strategies. One potential remedy is to include learning-strategy training into students' educational experiences. A major challenge, however, is that it has proven difficult to design training protocols that support students' self-regulation and transfer of effective learning strategies across a range of content. In this article we propose a practical theoretical framework called the knowledge, belief, commitment, and planning (KBCP) framework for guiding strategy training to promote students' successful self-regulation of effective learning strategies. The KBCP framework rests on the assumption that four essential components must be included in training to support sustained strategy self-regulation: (a) acquiring knowledge about strategies, (b) belief that the strategy works, (c) commitment to using the strategy, and (d) planning of strategy implementation. We develop these assumptions in the context of pertinent research and suggest that each component alone is not sufficient to promote sustained learning-strategy self-regulation. Our intent in developing this learning-strategy training framework is to stimulate renewed interest and effort in investigating how to effectively train learning strategies and their self-regulation and to guide systematic research and application in this area. We close by sketching an example of a concrete training protocol based on the KBCP framework.

Prospective Memory: Age related change is influenced by APOE genotype.

Non-focal prospective memory (PM) is sensitive to age-related decline; an additional impairment in focal PM is characteristic of mild stage Alzheimer's disease. This research explored whether, by mid-adulthood, the distinct demands of focal and non-focal PM expose differences in carriers of an APOE ε4 allele, a genetic risk factor for Alzheimer's disease. Thirty-three young and 55 mid-age adults, differentiated by APOE genotype, completed a category-decision task with a concurrent focal or non-focal PM demand. Only mid-age ε4 carriers showed a cost of carrying a focal PM intention. In addition, mid-age ε4 carriers showed a significantly greater cost of carrying a non-focal PM intention than young ε4 carriers, supporting a profile of accelerated aging. Consistency in the profile of cost differences observed in mid-age ε4 carriers and pathological aging may indicate premature vulnerability. Future research correlating a shift in PM performance with early genotype differences in brain-based markers of decline is important.

Effects of specific-level versus broad-level training for broad-level category learning in a complex natural science domain.

Category learning is a core component of course curricula in science education. For instance, geology courses teach categorization of rock types. Using the educationally authentic rock categories, the current project examined whether category learning at a broad-level (BL; igneous, sedimentary, and metamorphic rocks) could be enhanced by learning category information at a more specific-level (SL; e.g., diorite under igneous, breccia under sedimentary, etc.). Experiments 1 and 2 showed that SL training was inferior to BL training when participants were required to respond at the BL regardless of whether BL and SL category labels were presented simultaneously during classification training or SL categories were learned initially followed by training on the specific-broad level name associations. However, Experiments 3 and 4 showed that SL training was as good as BL training when the training was more extensive and participants were allowed to respond at the trained level. By considering confusion matrices (i.e., probabilities that instances in a given category was erroneously classified as belonging to other categories), we conjectured that between-SL category similarity, specifically the degree to which similar-looking SL categories belong to the same BL category, is an important factor in determining the efficacy of SL training. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Individual and age differences in block-by-block dynamics of category learning strategies.

The present experiment examined individual and age differences in the dynamics of category learning strategies. Participants learned categories determined by a disjunctive rule with relational features through a feedback training procedure. During training, participants responded to strategy probes following each block to provide online assessment of the extent to which rule- and exemplar-based strategies were used throughout the training period. We introduced this measure as an alternative to model-based approaches to assessing individual differences in strategy use during training. Following training, participants classified ambiguous transfer objects that were assumed to distinguish between earlier use of rule- and exemplar-based learning strategies. We included this measure to obtain a relatively objective index of strategy use during training. Next, participants provided global ratings of their use of rule- and exemplar-based strategies during training. Results showed that strategy preferences expressed on the final training block predicted categorisation of ambiguous transfer objects better than global strategy reports. In addition, we utilised the block-by-block strategy reports to investigate the dynamics of learners' strategy preferences over the course of training. The findings revealed greater fluidity in strategy preferences for both younger and older adults than has been previously documented in the category learning literature. The novel block-by-block strategy reports in conjunction with the transfer-based approach allowed for a more nuanced examination of individual and age differences in strategy use and categorisation performance.

Learning hierarchically organized science categories: simultaneous instruction at the high and subtype levels.

BACKGROUND: Most science categories are hierarchically organized, with various high-level divisions comprising numerous subtypes. If we suppose that one's goal is to teach students to classify at the high level, past research has provided mixed evidence about whether an effective strategy is to require simultaneous classification learning of the subtypes. This past research was limited, however, either because authentic science categories were not tested, or because the procedures did not allow participants to form strong associations between subtype-level and high-level category names. Here we investigate a two-stage response-training procedure in which participants provide both a high-level and subtype-level response on most trials, with feedback provided at both levels. The procedure is tested in experiments in which participants learn to classify large sets of rocks that are representative of those taught in geoscience classes. RESULTS: The two-stage procedure yielded high-level classification performance that was as good as the performance of comparison groups who were trained solely at the high level. In addition, the two-stage group achieved far greater knowledge of the hierarchical structure of the categories than did the comparison controls. CONCLUSION: In settings in which students are tasked with learning high-level names for rock types that are commonly taught in geoscience classes, it is best for students to learn simultaneously at the high and subtype levels (using training techniques similar to the presently investigated one). Beyond providing insights into the nature of category learning and representation, these findings have practical significance for improving science education.

Online and Clicker Quizzing on Jargon Terms Enhances Definition-Focused but Not Conceptually Focused Biology Exam Performance.

Mastery of jargon terms is an important part of student learning in biology and other science, technology, engineering, and mathematics domains. In two experiments, we investigated whether prelecture quizzes enhance memory for jargon terms, and whether that enhanced familiarity can facilitate learning of related concepts that are encountered during subsequent lectures and readings. Undergraduate students enrolled in neuroanatomy and physiology courses completed 10-minute low-stakes quizzes with feedback on jargon terms either online (experiment 1) or using in-class clickers (experiment 2). Quizzes occurred before conventional course instruction in which the terms were used. On exams occurring up to 12 weeks later, we observed improved student performance on questions that targeted memory of previously quizzed jargon terms and their definitions relative to questions on terms that were not quizzed. This pattern occurred whether those questions were identical (experiment 1) or different (experiment 2) from those used during quizzing. Benefits of jargon quizzing did not consistently generalize, however, to exam questions that assessed conceptual knowledge but not necessarily jargon knowledge. Overall, this research demonstrates that a brief and easily implemented jargon-quizzing intervention, deliverable via Internet or in-class platforms, can yield substantial improvements in students' course-relevant scientific lexica, but does not necessarily impact conceptual learning.

Toward a better understanding of costs in prospective memory: A meta-analytic review.

In prospective memory (PM) research, a common finding is that people are slower to perform an ongoing task with concurrent PM demands than to perform the same task alone. This slowing, referred to as costs, has been seen as reflecting the processes underlying successful PM. Historically, costs have been interpreted as evidence that attentional capacity is being devoted toward detecting PM targets and maintaining the intention in working memory; in other words, the claim is that participants are monitoring. A new account, termed delay theory, instead suggests that costs indicate a strategic speed/accuracy adjustment in favor of accuracy, allowing more time for PM-related information to reach its own threshold. Taking a meta-analytic approach, we first review studies in the PM literature that have reported ongoing task performance, both with and without a concurrent PM task, identifying key factors suitable for the meta-analysis. Next, we analyze the data of these studies, using our factors as moderators in a series of metaregressions, to determine their impact on the presence or magnitude of PM-related costs. Finally, we interpret the results of the meta-analysis from both monitoring and delay perspectives in an effort to better understand the nature of costs and what they reflect about the underlying cognitive processes involved in PM. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Adding the keyword mnemonic to retrieval practice: A potent combination for foreign language vocabulary learning?

The keyword mnemonic and retrieval practice are two cognitive techniques that have each been identified to enhance foreign language vocabulary learning. However, little is known about the use of these techniques in combination. Previous demonstrations of retrieval-practice effects in foreign language vocabulary learning have tended to use several rounds of retrieval practice. In contrast, we focused on a situation in which retrieval practice was limited to twice per item. For this situation, it is unclear whether retrieval practice will be effective relative to restudying. We advance the view that the keyword mnemonic catalyzes the effectiveness of retrieval practice in this learning context. Experiment 1 (48-h delay) partially supported this view, such that there was no testing effect with retrieval practice alone, but the keyword-retrieval combination did not promote better retention than keyword alone. Experiments 2 and 3 (1-week delay) supported the catalytic view by showing that the keyword-retrieval combination was better than keyword alone, but in the absence of keyword encoding there was no retrieval practice effect (replicating Experiment 1). However, with four rounds of retrieval practice, a marginally significant testing effect emerged (Experiment 3). Moreover, the routes through which participants reached each answer were identified by asking retrieval-route questions in Experiments 2 and 3. Keyword-mediated retrieval, which was observed sometimes even in no-keyword instructed conditions, was shown to be more effective than unmediated retrieval. Our findings suggest that incorporating effective encoding techniques prior to retrieval practice could augment the effectiveness of retrieval practice, at least for vocabulary learning.

Retrieval in prospective memory: Multiple processes or just delay?

In prospective memory (PM) research, a common finding is that PM accuracy is greater using focal, rather than nonfocal, cues. Under the multiprocess framework, the high PM performance for focal cues (cues that facilitate noticing of the target), often in the absence of task interference, reflects people's ability to rely on spontaneous retrieval processes. By contrast, nonfocal cues (cues that do not facilitate noticing) require monitoring. A competing explanation suggests that a single process underlies focal versus nonfocal PM: People adjust their delay in ongoing responding to allow enough time for PM information to reach awareness (delay theory). Participants' lower nonfocal performance arises because they fail to delay responding to a sufficient degree; with focal cues, the PM information accumulation rate is fast enough that no delay is necessary (and thus most everyone performs well). We sought to improve nonfocal PM performance by pairing a PM task with fast information accumulation to an ongoing task for which the requisite information accumulated more slowly. Reasoning from delay theory, we expected PM accuracy levels in this nonfocal PM task to approximate that observed in a focal PM task (for which the PM tasks were identical). In contrast to this expectation, the focal condition displayed significantly higher PM accuracy (despite demonstrating a reliably shorter response delay). In light of these findings, we concluded that the multiprocess interpretation is favoured.

Participation in Voluntary Re-quizzing Is Predictive of Increased Performance on Cumulative Assessments in Introductory Biology.

Low-stakes testing, or quizzing, is a formative assessment tool often used to structure course work. After students complete a quiz, instructors commonly encourage them to use those quizzes again to retest themselves near exam time (i.e., delayed re-quizzing). In this study, we examine student use of online, ungraded practice quizzes that are reopened near exam time after a first graded attempt 1-3 weeks prior. We find that, when controlling for preparation (performance in a previous science, technology, engineering, and mathematics [STEM] course and incoming biology knowledge), re-quizzing predicts better performance on two cumulative exams in introductory biology: a course posttest and final exam. Additionally, we describe a preliminary finding that, for the final exam, but not the posttest, re-quizzing benefits students with lower performance in a previous STEM course more than their higher-performing peers. But unfortunately, these struggling students are also less likely to participate in re-quizzing. Together, these data suggest that a common practice, reopening quizzes for practice near exam time, can effectively benefit student performance. This study adds to a growing body of literature that suggests quizzing can be used as both an assessment tool and a learning tool by showing that the "testing effect" extends to delayed re-quizzing within the classroom.

Event versus activity-based cues and motivation in school-related prospective memory tasks.

Prospective memory (PM), the ability to remember an intention in the future, is essential to children's everyday lives. We explored age differences (6- to 7- vs. 10- to 11-year-olds) in PM depending on the nature of the task and the children's motivation. Children performed event-based PM tasks (in which the cue was presented during the ongoing activity) and activity-based PM tasks (in which the cue consisted of finishing the ongoing activity). Additionally, the children were assigned to either a reward condition or a no-reward condition. The results showed better performance in event than in activity based tasks, with older children outperforming younger children in both. There was a marginal effect of reward for PM accuracy. These patterns suggest that the cue detection process and children's motivation play a role in PM performance during development.

Hey buddy, why don't we take it outside: An experience sampling study of prospective memory.

Relatively little research has focused on how prospective memory (PM) operates outside of the laboratory, partially due to the methodological problems presented by naturalistic memory research in general and by the unique challenges of PM in particular. Experience sampling methods (ESM) offer a fruitful avenue for this type of research, as recent work from Gardner and Ascoli (Psychology and Aging, 30, 209-219, 2015) has shown. They found that people thought about PM around 15% of the time, and that future thinking was more common than past thinking. In two studies, we replicated our own findings and those reported by Gardner and Ascoli. To summarize, people think about the future more often than the past (30% compared to 13%), and PM occupies our thoughts approximately 13-15% of the time, supporting claims made by some researchers that our episodic memory systems are forward-looking (Klein in Journal of Applied Research in Memory and Cognition, 2, 222-234, 2013). Of those PM thoughts, participants reported that 61% were internally cued, rather than externally triggered. Through the use of multi-level modeling, we additionally found that PM thoughts were more likely when the respondant was alone than with people, and earlier in the day. Finally, we found that participants higher in neuroticism were more likely to report thinking of PM, and that this was driven entirely by the anxiety facet. Most generally, we hope to have demonstrated the value of ESM to help researchers investigate and understand naturalistic PM.

Feature highlighting enhances learning of a complex natural-science category.

Learning naturalistic categories, which tend to have fuzzy boundaries and vary on many dimensions, can often be harder than learning well defined categories. One method for facilitating the category learning of naturalistic stimuli may be to provide explicit feature descriptions that highlight the characteristic features of each category. Although this method is commonly used in textbooks and classrooms, theoretically it remains uncertain whether feature descriptions should advantage learning complex natural-science categories. In three experiments, participants were trained on 12 categories of rocks, either without or with a brief description highlighting key features of each category. After training, they were tested on their ability to categorize both old and new rocks from each of the categories. Providing feature descriptions as a caption under a rock image failed to improve category learning relative to providing only the rock image with its category label (Experiment 1). However, when these same feature descriptions were presented such that they were explicitly linked to the relevant parts of the rock image (feature highlighting), participants showed significantly higher performance on both immediate generalization to new rocks (Experiment 2) and generalization after a 2-day delay (Experiment 3). Theoretical and practical implications are discussed. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Model-guided search for optimal natural-science-category training exemplars: A work in progress.

Under the guidance of a formal exemplar model of categorization, we conduct comparisons of natural-science classification learning across four conditions in which the nature of the training examples is manipulated. The specific domain of inquiry is rock classification in the geologic sciences; the goal is to use the model to search for optimal training examples for teaching the rock categories. On the positive side, the model makes a number of successful predictions: Most notably, compared with conditions involving focused training on small sets of training examples, generalization to novel transfer items is significantly enhanced in a condition in which learners experience a broad swath of training examples from each category. Nevertheless, systematic departures from the model predictions are also observed. Further analyses lead us to the hypothesis that the high-dimensional feature-space representation derived for the rock stimuli (to which the exemplar model makes reference) systematically underestimates within-category similarities. We suggest that this limitation is likely to arise in numerous situations in which investigators attempt to build detailed feature-space representations for naturalistic categories. A low-parameter extended version of the model that adjusts for this limitation provides dramatically improved accounts of performance across the four conditions. We outline future steps for enhancing the current feature-space representation and continuing our goal of using formal psychological models to guide the search for effective methods of teaching science categories.